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1.  INTRODUCTION 

In  its  role  as  an  agent  for  improving  software  technology  use  within  the  U.S.  Air  Force,  the  Software 
Technology  Support  Center  (STSC)  is  supporting  metrics  technology  improvement  activities  for  its 
customers.  These  activities  include:  disseminating  information  regarding  the  U.S.  Air  Force  Policy  on 
software  metrics  [AP93M-017],  providing  metrics  information  to  the  public  through  CrossTalk, 
conducting  customer  workshops  in  software  metrics,  guiding  metrics  technology  adoption  programs  at 
customer  locations,  researching  new  and  evolving  metrics  methodologies,  etc. 

Helping  customers  become  proficient  in  developing  and  using  software  metrics  to  support  their  software 
development  and/or  management  activities  is  crucial  to  customer  success.  The  STSC  metrics  support 
activities  must  be  tailored  to  the  customer‘s  needs  to  ensure 

a.  that  the  activities  are  appropriate  to  the  customer’s  organization  and  metrics  capability 
maturity,  and^ 

b.  that  the  customer  is  ready  to  make  improvements  based  on  the  support  obtained. 

Customer  support  needs  include  activities  based  on  their  apparent  metrics  capability  and  those  that  are 
particularly  focused  on  dealing  with  the  organizational  and  cultural  issues  that  often  need  to  be  addressed 
to  facilitate  change. 

This  guide  covers  the  following: 

a.  It  defines  a  metrics  capability  evaluation  method  that  deals  specifically  with  defining  a 
customer’s  metrics  capability. 

b.  It  presents  metrics  capability  questionnaires  that  help  gather  metrics  capability  data. 

c.  It  outlines  a  metrics  capability  evaluation  report  that  provides  the  basis  for  developing  a 
metrics  customer  project  plan. 

d.  It  provides  a  metrics  customer  profile  form  used  to  determine  the  initial  information 
required  to  prepare  for  a  metrics  capability  evaluation. 

e.  It  provides  a  customer  organization  information  form  that  helps  guide  the  STSC  in 
gathering  cultural  information  about  the  organization  that  will  help  with  developing  and 
implementing  the  metrics  customer  project  plan. 

2.  EVALUATION  APPROACH 

2.1  Background 

The  foundation  for  the  evaluation  method  is  "A  Method  for  Assessing  Software  Measurement 
Technology.” [DASK90]^  Metrics  capability  maturity  consists  of  5  maturity  levels  that  are  analogous  to 


*  Metrics  capability  maturity  (or  metrics  capability)  refers  how  well  an  organization  uses  metrics  to 
help  manage  and  control  project  performance,  product  quality,  and  process  implementation  and 
improvement.  This  concept  is  discussed  in  more  detail  in  [DASK90]. 
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the  software  Capability  Maturity  Model  (CMM)  levels  defined  by  the  Software  Engineering  Institute 
(SEI).[PAUL93]  This  guide  has  been  designed  to  cover  metrics  capability  maturity  Levels  1  through  3. 
When  metrics  capability  evaluations  show  a  strong  percentage  (e.g.,  25  percent  or  more)  of  organizations 
at  metrics  capability  maturity  Level  3,  the  scope  of  the  evaluation  (and  this  guide)  will  be  expanded  to 
cover  metrics  capability  maturity  Levels  4  and  5. 

This  guide  defines  a  set  of  questions  to  elicit  information  that  will  help  characterize  an  organization's 
metrics  capability.  The  themes  used  in  the  questionnaire  and  their  relationships  to  an  organization’s 
metrics  capability  maturity  (for  Levels  1  through  3)  are  shown  in  Appendix  A. 

The  guide  contains  two  metrics  capability  questionnaires  (one  for  acquisition  organizations  and  one  for 
software  development/maintenance  organizations).  The  questions  in  the  questionnaires  are  used  as  the 
basis  for  interviews  with  an  organization's  representative(s)  to  help  determine  their  metrics  capabOity 
maturity.  After  the  interviews  are  complete,  the  results  are  collated  and  reported  in  a  evaluation  report 
that  is  delivered  to  the  evaluated  organization.  Additional  work  with  the  evaluated  organization  will 
depend  on  the  organization’s  needs.  Section  2.2  discusses  the  evaluation  process.  Appendix  B  contains  a 
brief  metrics  customer  profile  form,  which  is  filled  out  as  a  precursor  to  the  metrics  capability  evaluation. 
Appendix  C  is  an  annotated  outline  of  the  metrics  capability  evaluation  report,  and  Appendix  D  contains 
the  customer  organization  information  form. 

2*2  Software  Metrics  Capability  Evaluation  Process 

The  software  metrics  capability  evaluation  process  consists  of  the  three  basic  parts: 

a.  An  initial  contact,  which  is  performed  when  it  is  determined  that  an  organization  needs 
and  wants  assistance  with  its  metrics  capability. 

b.  The  evaluation  interview,  which  is  the  central  activity  in  the  software  metrics  capability 
evaluation  process. 

c.  Collating  and  analyzing  the  results,  which  are  the  transition  activities  that  occur 
between  the  evaluation  interview  and  evaluation  follow-up. 

These  sets  of  activities  are  discussed  in  Paragraphs  2.2.1  through  2.2.3. 

In  addition  to  evaluation,  there  may  be  follow-up  activities.  These  include  more  detailed  work  with  the 
customer  that  will  provide  a  metrics  capability  improvement  strategy  and  plan  when  applicable. 

Paragraph  2.3  discusses  the  follow-up  activities. 

2.2.1  Initial  Contact 

The  initial  contact  with  a  customer  generally  is  set  up  through  an  STSC  customer  consultant.  The 
customer  consultant  briefs  an  assigned  member  of  the  STSC  metrics  team  regarding  a  customer’s  need  for 
a  metrics  capability  evaluation  and  provides  a  contact  for  the  metrics  team  member  at  the  customer’s  site. 

The  metrics  team  member  contacts  the  customer  by  phone  to  gain  an  initial  understanding  of  the 
customer’s  organization  and  to  set  up  the  evaluation  interview.  The  metrics  customer  profile  form  is  used 


^  The  assessment  method  defined  in  [DASK90]  was  based  on  the  Software  Engineering  Institute  (SEI) 
process  assessment  methodology,  which  is  currently  exemplified  in  the  Capability  Maturity  Model 
(CMM)  for  Software,  Version  1.1.  [PAUL93] 
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to  help  gather  that  information.  Information  collected  during  this  initial  contact  will  be  used  to  help 
determine  the  proper  approach  for  the  introduction  briefing  presented  during  the  evaluation  interview 
visit.  Only  the  point  of  contact  information  must  be  completed  at  this  time;  however,  it  is  highly 
desirable  to  include  the  STSC  business  information.  When  the  profile  is  not  completed  during  the  initial 
contact,  it  needs  to  be  completed  prior  to  (or  as  an  introduction  to)  the  evaluation  interview  at  the 
customer's  site. 

2.2.2  Evaluation  Interview 

Two  STSC  metrics  team  members  conduct  the  interviews  as  a  metrics  evaluation  team.  On  the  same  day 
as  the  evaluation  interview,  an  introduction  briefing  is  provided  to  key  people  within  the  organization  (to 
be  determined  jointly  by  the  evaluation  team  members,  the  customer  consultant  assigned  to  the 
organization,  and  the  organization’s  primary  point  of  contact).  The  purpose  of  the  briefing  is  to  manage 
customer  expectations.  This  is  accomplished,  in  part,  by  providing  education  with  respect  to: 

a.  The  concepts  of  metrics  maturity. 

b.  The  approach  of  the  metrics  evaluation  team. 

c.  What  to  expect  when  evaluation  results  are  provided. 

The  interviews  are  conducted  with  the  manager  most  closely  associated  with  the  software  development 
activities  for  the  program  (or  project)  under  question.^  One  other  representative  from  the  program  (or 
project)  should  participate  in  the  interview  (a  staff  member  responsible  for  metrics  analysis  and  reporting 
would  be  most  appropriate).  The  first  part  of  the  interview  is  to  complete  the  metrics  customer  profile. 
When  this  is  completed,  the  metrics  capability  questionnaire  most  related  to  the  organization  (either 
acquirer  or  development/maintenance  organization)  is  used  as  the  input  to  the  remainder  of  the  evaluation 
process.  The  questionnaire  sections  for  both  Levels  2  and  3  are  used  regardless  of  the  customer's 
perceived  metrics  capability. 

The  questions  in  the  metrics  capability  evaluation  questionnaires  have  been  formalized  to  require  answers 
of  yes,  no,  not  applicable  (NA),  or  don’t  know  (?).  If  an  answer  is  yes,  the  customer  needs  to  relate 
examples  or  otherwise  prove  performance  that  fulfills  the  question.  If  the  answer  is  no,  comments  may  be 
helpful  but  are  not  required.  (If  the  answer  is  don't  know,  a  no  answer  is  assumed.)  If  the  answer  is  NA 
and  it  can  be  shown  to  be  NA,  the  question  is  ignored  and  the  answer  is  not  counted  as  part  of  the  score. 
The  chosen  metrics  capability  evaluation  questionnaires  need  to  be  completed  before  the  interview  is 
considered  complete. 

An  evaluation  interview  should  not  take  more  than  one  day  for  one  program  (or  software  project).  If  an 
organization  is  to  be  assessed,  a  representative  sample  of  programs  (or  software  projects)  need  to  be 
assessed  and  each  requires  a  separate  interview. 


^  In  the  case  of  the  acquirer,  this  will  be  the  individual  responsible  for  overseeing  the  software 

development  organization.  In  the  case  of  a  development  or  maintenance  organization,  this  will  be  the 
software  project  manager. 
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2.2.3  Collating  and  Analyzing  the  Results 

The  metrics  capability  questionnaires  completed  during  the  interview(s)  and  their  associated  examples  (or 
other  evidence  of  metrics  capability  maturity,  see  Paragraph  B.l)  are  collated  and  returned  to  STSC  for 
analysis.  The  metrics  capability  evaluation  team  that  conducted  the  interview(s)  is  responsible  for 
analyzing  and  reporting  the  results.  An  assessed  program  (or  software  project)  is  at  Level  2  if  at  least 
80%  of  all  Level  2  questions  are  answered  yes.  Otherwise  the  organization  is  at  Level  1,  etc.  [DASK90] 
(Scoring  is  discussed  in  more  detail  in  Paragraph  B.l.  The  contents  of  the  metrics  capability  evaluation 
report  are  outlined  in  Appendix  C.) 

The  questions  in  the  metrics  c^ability  questionnaires  are  organized  by  metrics  capability  maturity  themes 
to  help  focus  the  interviews  and  the  results  analysis.  (The  themes,  as  defined  in  [DASK90],  and  their 
characteristics  at  metrics  capability  maturity  Levels  2  and  3  are  reported  in  Appendix  A.)  The  customer’s 
strengths  and  weaknesses  can  be  addressed  directly  with  the  information  gathered  during  the  interview 
session(s).  In  addition,  activities  for  becoming  more  effective  in  implementing  and  using  metrics  can  be 
highlighted  in  the  metrics  capability  evaluation  report  and  in  the  project  plan. 

2.3  Software  Metrics  Capability  Evaluation  Follow-up 

Software  metrics  capability  evaluation  follow-up  includes  two  sets  of  activities: 

a.  The  metrics  capability  evaluation  report. 

b.  The  project  plan  and  implementation. 

The  report  details  the  evaluation  results  and  provides  recommendations  for  an  initial  set  of  improvement 
activities. 

The  project  plan  consists  of  a  customer  approved,  detailed  plan  to  improve  the  customer’s  metrics 
capability  (which  may  include  other  aspects  of  support  to  the  customer  such  as  software  process  definition, 
project  management  support,  or  requirements  management  workshops,  etc.). 

The  customer’s  organizational  culture  is  important  in  developing  the  content  and  phasing  of  the  project 
plan.  Issues  such  as  ability  to  incorporate  change  into  the  organization,  management  commitment  to 
software  technology  improvement,  etc.,  often  need  to  be  addressed  in  developing  a  success-oriented  plan.'^ 

Metrics  capability  improvement  implementation  consists  of  the  physical  implementation  of  the  project 
plan  and  a  periodic  evaluation  of  the  customer’s  status  to  determine  the  program's  improvement  and  any 
required  modifications  to  the  plan.  The  project  plan  and  implementation  are  described  in  Paragraph 
2.3.2. 


Appendix  D  contains  an  organization  information  form  the  STSC  uses  to  help  define  cultural  issues 
that  need  to  be  addressed  in  the  project  plan. 
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2.3.1  Metrics  Capability  Evaluation  Report 

The  metrics  capability  evaluation  report  consists  of  two  parts: 

a.  The  analyzed  results  of  the  evaluation. 

b.  Recommendations  for  a  set  of  activities  that  will  help  improve  the  customer’s  metrics 
capability. 

The  results  portion  of  the  report  is  organized  to  discuss  the  customer’s  overall  software  metrics  capability 
and  to  define  the  areas  of  strengths  and  weaknesses  based  on  each  of  the  measurement  themes.  The 
recommendations  portion  of  the  report  describes  an  overall  improvement  strategy  that  provides  a  balanced 
approach  toward  metrics  c^ability  improvement  based  on  the  customer’s  current  evaluation  results. 
Appendix  C  contains  an  annotated  outline  of  the  report. 

2.3.2  Project  Plan  and  Implementation 

If  a  customer  has  the  interest  to  proceed  with  a  project  plan,  the  STSC  will  develop  the  plan  in 
conjunction  with  the  customer.  The  contents  of  the  project  plan,  the  estimates  for  plan  implementation, 
and  the  schedule  will  be  developed  specifically  for  each  customer's  needs.  Due  to  the  possible  variations 

in  customer  needs,  it  is  difficult  to  determine  the  exact  contents  of  the  plan.  At  a  minimum,  the  project 

plan  contains  the  following  information: 

a.  An  executive  overview,  which  includes  a  synopsis  of  the  customer’s  current  software 
metrics  capability  maturity  and  a  general  outline  of  the  plan  to  be  implemented. 

b.  Organizational  responsibilities  for  the  customer,  the  customer’s  interfacing 
organizations  (e.g.,  a  contractor),  and  the  STSC.  Issues  that  arise  based  on 
organizational  information  are  highlighted. 

c.  Improvement  objectives. 

d.  A  set  of  activities  to  support  improvement  [e.g.,  a  Work  Breakdown  Structure  (WBS)] 
and  a  description  of  the  activities’  interrelationships. 

e.  A  schedule  for  implementation  and  for  periodic  evaluation  of  the  customer’s  progress. 
(The  periodic  evaluation  may  be  implemented  as  additional  metrics  capability 
evaluations,  as  described  in  this  guide.) 

f.  Effort  and  cost  estimates  for  STSC  support. 

g.  Facility  requirements  for  training  and  other  activities. 

h.  Descriptions  of  STSC  products  to  be  delivered  as  part  of  the  improvement 
implementation. 

After  the  plan  is  approved,  the  metrics  capability  improvement  implementation  follows  the  plan.  The 
periodic  evaluations  of  the  customer's  products  provide  feedback  regarding  the  customer’s  progress  and  an 
opportunity  to  revise  the  plan  if  the  improvement  is  not  proceeding  according  to  the  plan.  In  this  way,  the 
plan  and  implementation  process  can  be  adjusted  as  necessary  to  support  the  customer’s  ongoing  needs. 
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APPENDIX  A.  MEASUREMENT  THEMES  AND  RELATIONSHIPS 

Table  A-1  shows  the  six  metrics  themes  and  relates  the  themes  to  software  metrics  capability  maturity 
Levels  1  through  3. 

Table  A-1.  Themes  and  Levels  of  Software  Metrics  Capability  Maturity.® 


Theme 

Initial 
(Level  1) 

Repeatable 
(Level  2) 

Defined 
(Level  3) 

1.  Formalization 
of  development 
process 

Process  unpredictable 

Project  depends  on  seasoned 
professionals 

No/poor  process  focus 

Projects  repeat  previously 
mastered  tasks 

Process  depends  on 
experienced  people 

Process  characterized  and 
reasonably  understood 

2.  Formalization 
of  metrics 
process 

Little  or  no  formalization 

Formal  procedures 
established 

Metrics  standards  exist 

Documented  metrics 
standards 

Standards  applied 

3.  Scope  of 
metrics 

Occasional  use  on  projects 
with  seasoned  people  or 
not  at  all 

Used  on  projects  with 
experienced  people 

Project  estimation 
mechanisms  exist 

Metrics  have  project  focus 

Goal/Question/Metric 
package  development 
and  some  use 

Data  collection  and 
recording 

Specific  automated  tools 
exist  in  the  environment 

Metrics  have  product  focus 

4.  Implementation 
support 

No  historical  data  or 
database 

Data  (or  database)  available 
on  a  per  project  basis 

Product-level  database 

Standardized  database  used 
across  projects 

5.  Metrics 
evolution 

Little  or  no  metrics 
conducted 

Project  metrics  and 

management  in  place 

Product-level  metrics  and 
management  in  place 

6.  Metrics  support 
for  mgmt 
control 

Management  not  supported 
by  metrics 

Some  metrics  support  for 
management 

Basic  control  of  | 

commitments  ' 

Product-level  metrics  and 
control 

®  The  information  in  this  table  has  been  extracted  directly  from  [DASK90]. 
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APPENDIX  B.  SOFTWARE  METRICS  CAPABILITY  QUESTIONNAIRES 

This  appendix  contains  scoring  information  for  the  software  metrics  capability  evaluations  along  with 
copies  of  the  metrics  customer  profile  form  and  the  two  software  metrics  capability  evaluation 
questionnaires. 

The  metrics  customer  profile  form  helps  gather  general  customer  information  for  choosing  the  metrics 
capability  evaluation  questionnaire  and  for  defining  the  contents  of  the  project  plan.  The  two  software 
metrics  capability  evaluation  questionnaires  are  as  follows: 

a.  An  acquisition  organization  questionnaire.  The  focus  of  this  questionnaire  is  to 
determine  the  metrics  capability  level  of  a  software  acquisition  organizations. 

b.  A  software  development/maintenance  organization  questionnaire.  The  focus  of  this 
questionnaire  is  to  determine  the  metrics  capability  level  of  software  development  or 
maintenance  organizations. 

B.l  Use  of  Questionnaires  and  Scoring 

B.1.1  Use  of  Questionnaires 

These  two  metrics  capability  evaluation  questionnaires  provide  the  contents  of  the  evaluation  interviews 
described  in  Paragraph  2.2.2.  The  questions  from  the  questionnaires  are  asked  as  written.  The  questions 
for  Levels  2  and  3  are  used  for  all  interviews.  The  comments  for  each  question  are  used  to  point  to 
examples  and  other  evidence  of  metrics  capability  maturity  based  on  the  activities  referred  to  in  the 
question.  The  answers  to  the  questions  and  the  examples  and  comments  are  the  inputs  to  the  scoring 
activity  presented  in  Paragraph  B .  1 .2. 

B.l. 2  Scoring 

Scoring  from  the  two  metrics  capability  evaluation  questionnaires  is  relatively  simple: 

a.  If  the  answer  to  a  question  is  yes,  then  proof  of  conformance  needs  to  be  shown  to  ensure 
that  the  customer  has  performed  the  activity(ies)  indicated  in  the  question.  Proof  of 
conformance  includes: 

1.  Metrics  standards  for  the  organization. 

2.  Software  acquisition  plans,  development  plans,  or  contract  statements  that 
incorporate  metrics  requirements. 

3.  Meeting  minutes  or  other  items  that  indicate  use  of  metrics. 

4.  Examples  of  database  outputs. 

5.  Concurrence  given  by  two  or  more  individuals  from  the  same  organization  who 
are  interviewed  separately. 

6.  Informal  notes. 

7.  Briefing  charts  from  management  evaluations. 

8.  etc. 

b.  If  the  answer  is  no,  or  don't  know,  then  the  answer  is  scored  as  no. 
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c.  If  the  answer  is  NA,  then  question  is  subtracted  from  the  total  number  of  questions  for 
that  maturity  level  and  the  answer  is  not  included  in  the  overall  score. 

d.  When  80%  or  more  of  the  Level  2  questions  are  answered  yes  (with  prooO,  then  the 
organization  is  considered  to  be  a  Level  2.  Otherwise  the  organization  is  considered  to 
be  a  Level  1. 

e.  If  the  organization  is  a  Level  2  and  also  answers  80%  or  more  of  the  Level  3  questions 
yes  (with  proof),  then  the  organization  is  considered  to  be  a  Level  3.  Otherwise,  the 
organization  is  considered  to  be  a  Level  1  or  2  as  indicated  in  Item  d. 

The  organization's  metrics  capability  level,  as  indicated  from  the  scoring  process,  the  proofs  of 
conformance,  and  comments  are  all  used  as  inputs  to  the  metrics  capability  evaluation  report.  Appendix 
C  contains  an  annotated  outline  of  a  metrics  capability  evaluation  report. 
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B,2  Metrics  Customer  Profile  Form 

1 .  Point  of  Contact  information: 

a.  Name:  _ _ _ _ _ _ 

b.  Position:  . . . . . 

c.  Office  symbol:  _ _ _ _ 

d.  Location:  _  _ _ 

e.  Phone#:  _  DSN:  _ 

f.  Fax  number:  _ _ _ 

g.  Email  address:  _ _ 

h.  Organization  name:  _ _ 

i.  Products:  _ _ _ 

2.  Environment  information: 

a.  Hardware  platform:  _ _ 

b.  Languages  used:  _ _ _ 

c.  Tools  used  for  metrics:  _ _ 

3.  Organization  information: 

a.  Major  command  (ACC,  AFMC,  AETC,  AMC,  other:  _ ) 

b.  Copy  of  organization  chart  (At  least  name  and  rank  of  commanding  officer): 


c.  Type(s)  of  software  (real  time,  communication,  command  &  control,  MIS,  other): 


d.  Type(s)  of  activity  (development,  acquisition,  maintenance,  combination,  other): 
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e.  Are  project  teams  comprised  of  members  from  more  than  one  organization?  (If  yes, 
please  give  examples) _ _ _ — 


f.  Typical  size  of  development  organization  for  a  particular  program  (or  project)  (less  than 
10, 10-40,  more  than  40  personnel):  _ _ _ 


g.  Typical  length  of  project  (<  6  mo,  6  - 18  mo,  18  mo  -  3  yr,  >  3  yr): 


4.  General  background: 

a.  What  are  the  organization’s  strengths? 


b.  Can  you  demonstrate  these  strengths  through  measurements  or  other  objective  means? 
(if  yes,  examples?):  _ _ _ 


c.  What  are  the  organization's  biggest  challenges? 


d.  Have  measurements  or  other  objective  means  been  used  to  understand  or  to  help  manage 
these  challenges?  (if  yes,  examples?):  _ _ 


5.  Metrics  background: 

a.  Does  your  organization  require  Software  Development  Plans  to  be  developed  and  used?  . 

b.  Are  project  management  tools  used?  (examples?):  _ 


c.  How  is  project  status  reported?  (examples?): 
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d.  How  is  product  quality  reported?  (examples?): 


e.  What  forces  are  driving  metrics  interest  in  your  organization  (SAF/AQ,  CO,  self,  etc.)?  . 


6.  STSC  business  information: 

a.  Has  the  organization  received  STSC  information  or  services? 

1.  CrossTalk?  _ _ _ _ 

2.  Technology  Reports?  _ _ 

3.  Workshops?  _ 

4.  Consulting? _ _ 

b.  Does  the  organization  need  help? _ 

c.  Does  the  organization  want  help?  _ 

d.  The  organization  would  like  help  with  (describe):  _ 

e.  How  well  is  the  organization  funded  for  new  technology  adoption  (including  training)? 

1.  Are  there  funds  to  pay  for  STSC  Products  and  Services? _ 

2.  Is  the  organization  willing  to  pay?  _ 

f.  Are  their  needs/wants  a  match  to  STSC  products  and  services? _ 
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B.3  Acquisition  Organization  Questionnaire^ 

B.3,1  Questions  for  Metrics  Capability  Level  2 

B3AA  Theme  1:  Formalization  of  Source  Selection  and  Contract  Monitoring  Process 

#  Question  Yes  No  NA 

la  Is  a  Software  Capability  Evaluation  (SCE)  or  Software  □  □  □ 

Development  Capability  Evaluation  (SDCE)  for  developers  part 
of  your  source  selection  process?^ 

Comments: 


lb  Is  proof  of  a  specific  CMM  Level  required  from  developers  as  D  H  D 

part  of  your  source  selection  process? 

Comments: 


2  Does  your  organization  require  and  evaluate  developers'  draft  □  □  □ 

software  development  plans  as  part  of  the  source  selection 
process? 

Comments: 


3  Are  software  metrics  required  as  part  of  developers'  software  □  □  □  □ 

development  plans  (or  other  contractually  binding  metrics 
plans)? 

Comments: 


^  Throughout  these  questionnaires,  acquirer  refers  to  an  organization  that  acquires  software  or  systems. 
Developer  refers  to  an  organization  that  develops  or  maintains  software  or  systems  for  an  acquirer. 
(For  example,  a  developer  could  refer  to  a  non-military  organization  (e.g.,  a  defense  contractor,  a 
university,  etc.)  that  works  under  the  terms  of  a  legal  contract;  an  external  Government  or  MUitary 
organization  that  works  under  the  terms  of  a  Memorandum  of  Agreement  (MO A);  or  an  organic 
organization  tasked  with  developing  or  maintaining  software  under  an  informal  agreement,  etc.) 
Contract  refers  to  an  agreement  between  the  acquirer  and  the  contractor,  regardless  of  its  actual  form 
(e.g.,  an  MO  A). 

^  Score  only  one  correct  for  a  yes  response  to  either  la  or  lb.  If  neither  is  a  yes  answer,  score  only  one 
no. 


13 


SW  Metrics  Capability  Evaluation  Guide 
Version  2.0,  October  20, 1995 


#  Question  Yes  No  NA  ? 

4  Are  softv/are  cost  and  schedule  estimates  required  from  the  □  □  □  □ 

developer  as  part  of  the  source  selection  process? 

Comments: 

5  Is  the  developer's  project  performance  monitored  based  on  the  □  □  □  □ 

cost  and  schedule  estimates? 

Comments: 

6  Are  the  acquirers’  management  plans  developed,  used,  and  □  □  □  □ 

maintained  as  part  of  managing  a  program? 

Comments: 


B3A.2  Theme  2:  Formalization  of  Metrics  Process 
#  Question  Yes  No  NA  ? 

1  Is  there  a  written  organizational  policy  for  collecting  and  □  □  □  □ 

maintaining  software  metrics  for  this  program? 

Comments: 

2  Is  each  program  required  to  identify  and  use  metrics  to  show  □  □  □  □ 

program  performance? 

Comments: 

3  Is  the  use  of  software  metrics  documented?  □  □  □  □ 

Comments: 
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#  Question  Yes  No  NA 

4  Are  developers  required  to  report  a  set  of  standard  metrics?  □  □  □ 

Comments: 


BJJJ  Theme  3:  Scope  of  Metrics 

#  Question  Yes  No  NA 

1  Are  internal  measurements  used  to  determine  the  status  of  the  □  □  □ 

activities  performed  for  planning  a  new  acquisition  program? 

Comments: 


2  Are  measurements  used  to  determine  the  status  of  software  □  □  □ 

contract  management  activities? 

Comments: 


3  Do(es)  your  contract(s)  require  metrics  on  the  developer's  actual  □  □ 

results  (e.g.,  schedule,  size,  and  effort)  compared  to  the 
estimates? 

Comments: 


4  Can  you  determine  whether  the  program  is  performing  □  □  □ 

according  to  plan  based  on  measurement  data  provided  by  the 
developer? 

Comments: 


5  Are  measurements  used  to  determine  your  organization’s  □  □  □ 

planned  and  actual  effort  applied  to  performing  acquisition 
planning  and  program  management? 

Comments: 
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#  Question  Yes  No  NA 

6  Are  measurements  used  to  determine  the  status  of  your  □  □  □ 

organization’s  software  configuration  management  activities? 

Comments: 


B.3JA  Theme  4:  Implementation  Support 

#  Question 

1  Does  the  program  (or  project)  have  a  database  of  metrics 
information? 

Conunents: 

2  Do  you  require  access  to  the  contractor’s  metrics  data  as  well  as  □  □  □ 

completed  metrics  reports? 

Comments: 

3  Does  your  database  (or  collected  program  data)  include  both  □  □  □ 

developer’s  and  acquirer’s  metrics  data? 

Comments: 


Yes  No  NA  ? 

□  □  □  □ 
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B3J,5  Theme  5:  Metrics  Evolution 

#  Question  Yes  No  NA 

1  Is  someone  from  the  acquisition  organization  assigned  specific  □  □  □ 

responsibilities  for  tracking  the  developer's  activity  status  (e.g., 
schedule,  size,  and  effort)? 

Comments: 


2  Does  the  developer  regularly  report  the  metrics  defined  in  the  CD  □ 
developer’s  software  development  plan  (or  other  contractually 
binding  metrics  plan)? 

Comments: 


3  Do  your  contracts  have  clauses  that  allow  the  acquirer  to  request  □ 
changes  to  the  developer's  metrics  based  on  program  needs? 

Conunents: 


B,3J.6  Theme  6:  Metrics  Support  for  Management  Control 

#  Question 

1  Do  you  track  your  developer's  performance  against  the 
developer’s  commitments? 

Comments: 

2  Are  the  developer's  metrics  results  used  as  an  indicator  of  when  □  □  □ 

contract  performance  should  be  analyzed  in  detail? 

Comments: 

3  Are  metrics  results  used  to  support  risk  management,  □  □  □ 

particularly  with  respect  to  cost  and  schedule  risks? 

Comments: 


Yes  No  NA  ? 

□  □  □  □ 
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Question  Yes 

Are  program  acquisition  and/or  program  management  metrics  □ 
used  to  help  determine  when  changes  should  be  made  to  your 
plans  (e.g.,  changes  to  schedules  for  completion  of  planning 
activities  and  milestones,  etc.)? 

Conunents: 


Are  measurements  used  to  determine  the  status  of  verification  &  □ 

validation  activities  for  software  contracts? 


Comments: 


SW  Metrics  Capability  Evaluation  Guide 
Version  2.0,  October  20, 1995 


B.3.2  Questions  for  Metrics  Capability  Level  3 

B.3.2A  Theme  1:  Formalization  of  Source  Selection  and  Contract  Monitoring  Process 

#  Question  Yes  No  NA 

1  Do  you  require  developers  to  show  proof  of  software  □  □  □ 

development  maturity  at  a  minimum  of  CMM  Level  3? 

Comments: 


2  Is  your  software  acquisition  process  reviewed  for  improvement  □  □ 

periodically? 

Comments: 

3  Does  your  organization  have  a  standard  software  acquisition  □  □ 

process? 

Comments: 


4  Do  one  or  more  individuals  have  responsibility  for  maintaining  □  □  □ 

the  organization’s  standard  software  acquisition  processes? 

Comments: 


5  Does  the  organization  follow  a  written  policy  for  developing  and  □ 
maintaining  the  acquisition  process  and  related  information 
(e.g.,  descriptions  of  approved  tailoring  for  standards  based  on 
program  attributes)? 

Comments: 
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B322  Theme  2:  Formalization  of  Metrics  Process 

#  Question  Yes 

1  Do  you  have  documented  standards  for  metrics  definitions  and  □ 
for  reporting  formats  you  require  from  developers? 

Comments: 


2  Are  these  standards  tailorable  to  the  size,  scope,  and  type  of  the 
software  to  be  acquired? 

Conunents: 


3  Are  specific  metrics  requested  for  each  new  acquisition  based  on 
your  organization’s  metrics  standards? 

Comments: 


4  Is  someone  from  your  organization  assigned  specific 

responsibilities  for  maintaining  and  analyzing  the  contractor's 
metrics  regarding  the  status  of  software  work  products  and 
activities  (e.g„  effort,  schedule,  quality)? 

Comments: 


B.3.2.3  Theme  3:  Scope  of  Metrics 

#  Question  Yes 

1  Do  you  collect,  maintain,  and  report  metrics  data  for  all  new  (in  □ 
the  last  3  years)  contracts? 


Comments: 
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Question  Yes 

Do  you  use  automated  tools  that  support  metrics  collection,  □ 

maintenance,  and  reporting? 

Comments: 

Do  you  and  your  developer(s)  use  automated  metrics  tools  that  Gl 
allow  you  to  share  contract  metrics  data? 

Comments: 

During  contract  negotiations,  do  the  program  goals  drive  the  □ 

metrics  required  for  the  contract? 

Comments: 

Do  the  metrics  collected  include  specific  product  metrics  (e.g.,  □ 

quality,  reliability,  maintainability)? 

Conunents: 

Do  you  require  metrics  summary  reports  that  show  general  □ 

program  trends  as  well  as  detailed  metrics  information? 


Comments: 
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BJ,2A  Theme  4:  Implementation  Support 

#  Question 

1  Does  your  program  metrics  database  include  information  on 
specific  product  metrics  (e.g.,  quality,  reliability, 
maintainability)? 

Comments: 


Yes  No  NA 
□  □  □ 


2  Do  you  share  metrics  data  across  programs? 
Comments: 


Is  the  metrics  data  shared  through  a  common  organizational 
database? 

Conunents: 


□ 


□ 


□ 


Does  your  organization  have  a  standard  length  of  time  that  you 
retain  metrics  data? 

Comments: 


Does  the  organization  verify  the  metrics  data  maintained  in  the 
metrics  database? 

Comments: 


□ 


Does  your  organization  manage  and  maintain  the  metrics 
database? 

Comments: 


□ 
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B.5.2.5  Theme  5:  Metrics  Evolution 

#  Question  Yes  No  NA  ? 

1  Do  you  use  product  metrics  in  making  management  decisions?  □  □  □  □ 

(e.g.,  a  decision  is  made  to  delay  schedule  because  of  known 

defects). 

Comments: 

2  Are  product  metrics  reported  during  program  management  □  □  □  □ 

reviews  (e.g.,  defects  by  severity,  or  defects  by  cause)? 

Comments: 

3  Are  both  project  and  product  metrics  used  in  making  □  □  □  □ 

management  decisions  regarding  contract  performance? 

Comments: 

4  Does  your  organization  review  the  current  metrics  set  □  □  □  □ 

periodically  for  ongoing  usefulness? 

Comments: 

5  Does  your  organization  review  the  current  metrics  set  □  □  □  □ 

periodically  to  determine  if  new  metrics  are  needed? 

Comments: 
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B3.2.6  Theme  6:  Metrics  Support  for  Management  Control 

#  Question  Yes  No  NA  ? 

1  Are  measurements  used  to  determine  the  status  of  the  program  □  □  □  □ 

office  activities  performed  for  managing  the  software 

requirements? 

Comments: 

2  Are  product  metrics  used  as  an  indicator  for  renegotiating  the  □  □  □  □ 

terms  of  contract(s)  when  necessary? 

Comments: 

3  Are  product  metrics  used  in  reports  forwarded  to  higher  level  □  □  □  □ 

management  concerning  contract  performance? 

Comments: 

4  Are  measurements  used  to  forecast  the  status  of  products  during  □  □  □  □ 

their  development? 

Comments: 

5  Are  product  metrics  used  as  inputs  to  award  fee  calculations  for  □  □  □  □ 

cost  plus  award  fee  contracts? 

Comments: 

6  Do  metrics  serve  as  inputs  for  determining  when  activities  need  □  □  □  □ 

to  be  initiated  (or  modified)  to  mitigate  technical  program  risks? 

Comments: 
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B.4  Software  Development/Maintenance  Organization  Questionnaire 
B.4.1  Questions  for  Metrics  Capability  Level  2 

B.4JJ  Theme  1:  Formalization  of  the  Development  Process 


#  Question 

la  Has  your  organization  been  assessed  via  the  SEI CMM?*  (This 
could  be  an  independent  assessment  or  an  internal  assessment 
supported  by  an  SEI  authorized  source). 


Yes 

No 

NA 

9 

□ 

□ 

□ 

□ 

Comments: 


lb  Has  your  organization  been  assessed  via  some  vehicle  other  than  □  □  □  □ 

the  SEI  CMM? 

Comments: 

2  Are  software  development  plans  developed,  used,  and  maintained  □  □  □  □ 

as  part  of  managing  software  projects? 

Comments: 

3  Are  software  metrics  included  in  your  software  development  □  □  □  □ 

plans  or  other  contractual  binding  document(s)? 

Comments: 


4  Does  your  organization  have  an  ongoing  software  process  □  □  □ 

improvement  program? 

Comments: 


Score  only  one  correct  for  a  yes  response  to  either  la  or  lb.  If  neither  is  a  yes  answer,  score  only  one 
no. 
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BAJ.2  Theme  2:  Formalization  of  Metrics  Process 

#  Question 

1  Is  there  a  written  policy  for  collecting  and  maintaining  project 
management  metrics  (e.g.  cost,  effort,  and  schedule)? 

Comments: 

2  Do  standards  exist  for  defining,  collecting,  and  reporting  □  □  □ 

metrics? 

Comments: 

3  Is  each  project  required  to  identify  and  use  metrics  to  show  □  □  □ 

project  performance? 

Comments: 


Yes  No  NA  ? 

□  □  □  □ 


BA.  1,3  Theme  3:  Scope  of  Metrics 

#  Question 

1  Are  measurements  used  to  determine  the  status  of  activities 
performed  during  software  planning? 

Comments: 

2  Are  measurements  used  to  determine  and  track  the  status  of  □  □  □  □ 

activities  performed  during  project  performance? 

Comments: 

3  Does  the  project  manager  establish  cost  and  schedule  estimates  □  □  □  □ 

based  on  prior  experience? 

Comments: 


Yes  No  NA  ? 

□  □  □  □ 
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BAA  A  Theme  4:  Implementation  Support 

#  Question 

1  Is  there  a  project  database  of  metrics  information? 
Comments: 


Yes  No  NA  ? 

□  □  □  □ 


2  Is  the  project  manager  reponsible  for  implementing  metrics  for  □ 
the  project? 

Comments: 


3  Do  you  keep  metrics  from  project  to  project  (historical  data)? 
Comments: 


□  □  □  □ 


BAA. 5  Theme  5:  Metrics  Evolution 

#  Question 

1  Do  you  report  the  project’s  actual  results  (e.g.,  schedule  and 
cost)  compared  to  estimates? 

Comments: 

2  Is  someone  on  the  staff  assigned  specific  responsibilities  for  □  □  □  □ 

tracking  software  project  activity  status  (e.g.,  schedule,  size, 

cost)? 

Comments: 

3  Do  you  regularly  report  the  metrics  defined  in  the  software  □  □  □  □ 

development  plan  or  other  contractually  required  document(s)? 

Comments: 


Yes 

No 

NA 

9 

□ 

□ 

□ 

□ 
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BA.L6  Theme  6:  Metrics  Support  for  Management  Control 

#  Question 

1  Do  metrics  results  help  the  project  manager  manage  deviations 
in  cost  and  schedule? 

Comments: 


Yes 

No 

NA 

9 

□ 

□ 

□ 

□ 

2  Are  measurements  used  to  determine  the  status  of  softv^are  □  □ 

configuration  management  activities  on  the  project? 

Comments: 

3  Are  measurements  used  to  determine  the  status  of  softv^are  □  □ 

quality  assurance  activities  on  the  project? 

Comments: 


4  Are  measurements  used  to  determine  the  status  of  the  activities  □  □  □ 

performed  for  managing  the  allocated  requirements  (e.g.,  total 
number  of  requirements  changes  that  are  proposed,  open, 
approved,  and  incorporated  into  the  baseline)? 

Comments: 


5  Are  cost  and  schedule  estimates  documented  and  used  to  refine  □  □  □  □ 

the  estimation  process? 

Comments: 


6  Do  you  report  metrics  data  to  the  customer  based  on  customer  □  □  □  □ 

requirements? 

Comments: 
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B.4.2  Questions  for  Metrics  Capability  Level  3 

BA2J  Theme  1:  Formalization  of  the  Development  Process 

#  Question  Yes 

1  Is  your  software  development  process  reviewed  for  improvement  □ 

periodically? 

Conunents: 

2  Does  your  organization's  standard  software  process  include  □ 

processes  that  support  both  software  management  and  software 
engineering? 

Comments: 

3  Are  your  processes  tailorable  to  the  size/scope  of  the  project?  □ 

Comments: 


B.4.22  Theme  2:  Formalization  of  Metrics  Process 
#  Question  Yes 

1  Do  you  have  documented  organizational  standards  for  metrics  □ 

(e.g.,  metrics  definitions,  analysis,  reports,  and  procedures)? 

Comments: 

2  Are  these  standards  tailorable  to  the  size  and  scope  of  the  □ 

software  project? 

Comments: 

3  Are  there  standards  established  for  the  retention  of  metrics?  □ 


Comments: 
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#  Question 

4  Are  specific  project  and  product  metrics  proposed  for  each 

software  project  based  on  the  organization’s  metrics  standards? 


Yes 

No 

NA 

7 

□ 

□ 

□ 

□ 

Comments: 


5  Is  someone  assigned  specific  responsibilities  for  maintaining  □ 

and  analyzing  metrics  regarding  the  status  of  software  work 
products  and  activities  (e.g.,  size,  effort,  schedule,  quality)? 

Comments: 


6  Does  the  organization  collect,  review,  and  make  available  □  □  □ 

information  related  to  the  use  of  the  organization’s  standard 
software  process  (e.g.,  estimates  and  actual  data  on  software 
size,  effort,  and  cost;  productivity  data;  and  quality 
measurements)? 

Comments: 


B.4.2,3  Theme  3:  Scope  of  Metrics 

#  Question  Yes  No  NA  ? 

1  Do  the  project/organization  management  and  technical  goals  drive  □  □  □  □ 

the  metrics  required? 

Comments: 

2  Do  you  collect,  maintain,  and  report  project  and  product  metrics  □  □  □  □ 

data  for  all  projects? 

Comments: 
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#  Question  Yes  No  NA 

3  Do  you  use  automated  tools  that  support  metrics  collection,  □  □  □ 

maintenance,  and  reporting? 

Conunents: 


4  Do  the  metrics  collected  include  specific  product  metrics  (e.g., 
quality,  reliability,  maintainability)? 

Comments: 


5  Do  you  report  product  metrics  (e.g.,  problem/defect  density  by 

product;  amount  of  rework;  and/or  status  of  allocated  requirements) 
throughout  the  development  life  cycle? 

Comments: 


B.4.2A  Theme  4:  Implementation  Support 


#  Question  Yes  No  NA 

1  Does  your  metrics  database  include  information  on  specific  product  □  □  □ 

metrics  (e.g.,  quality,  reliability,  maintainability)? 

Comments: 


2  Do  you  share  metrics  data  across  software  projects? 
Comments: 


□  □  □  □ 


3  Is  the  metrics  data  shared  through  a  common  organizational  □ 

database? 

Comments: 
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#  Question 

4  Does  your  organization  have  a  standard  length  of  time  that  you 
retain  metrics  data? 

Comments: 


Yes 

No 

NA 

•> 

□ 

□ 

□ 

□ 

5  Does  your  organization  verify  the  metrics  data  maintained  in  the 
metrics  database? 

Comments: 


6 


Does  your  organization  manage  and  maintain  the  metrics  database?  □  □ 


□  □ 


Comments: 


7  Have  normal  ranges  been  established  for  project  metrics  reported  □  □ 

(e.g.,  the  difference  between  planned  and  actual  schedule 
commitments)? 

Comments: 


BA, 2, 5  Theme  5:  Metrics  Evolution 

#  Question  Yes  No  NA 

1  Do  you  use  product  metrics  as  well  as  project  metrics  in  making  □  □  □ 

management  decisions? 

Comments: 

2  Are  product  metrics  as  well  as  project  metrics  reported  during  □  □  □ 

program  management  reviews  (e.g.,  the  number  of  defects  per 

SLOC)? 

Comments: 
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#  Question 

3  Do  you  report  metrics  to  your  internal  manager? 
Comments: 

4  Do  you  report  metrics  to  your  customer? 
Comments: 


Yes  No  NA  ? 
□  □  □  □ 


□  □  □  □ 


BA.2,6  Theme  6:  Metrics  Support  for  Management  Control 

#  Question  Yes  No  NA 

1  Are  product  metrics  as  well  as  project  metrics  used  as  indicators  for  □  □  □ 

renegotiating  the  terms  of  contract(s)  when  necessary  (e.g.,  you 
decide  to  extend  a  schedule  based  on  the  known  number  of  defects 
in  the  product)? 

Conunents: 


2  Do  metric  results  help  isolate  technical  problems?  □  □  □  □ 

Comments: 


3  Are  improvements  to  the  metrics  process  (including  metrics  □  □  □ 

standards,  procedures,  definitions,  etc.)  based  on  analysis  and 

lessons  learned? 

Comments: 

4  Are  measurements  used  to  determine  the  quality  of  the  software  □  □  □ 

products  (i.e.,  numbers,  types,  and  severity  of  defects  identified)? 

Comments: 
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Question 

Do  you  maintain  metrics  specifically  to  help  you  manage  your 
project? 

Comments: 


Are  management  decisions  made  as  a  result  of  metrics  reported 
(e.g.,  is  corrective  action  taken  when  actual  results  deviate 
significantly  from  the  project’s  software  plans)? 

Comments: 


Are  metrics  that  are  reported  to  the  customer  consistent  with 
internally  reported  metrics? 


Comments: 
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APPENDIX  C.  SOFTWARE  METRICS  CAPABILITY  EVALUATION  REPORT: 

ANNOTATED  OUTLINE 

The  goals  of  the  software  metrics  capability  evaluation  report  are  as  follows: 

a.  Report  the  results  of  the  evaluation.  The  results  have  two  components: 

1.  General  results  (i.e.,  metrics  capability  Level  and  an  overview  of  the 
organization's  metrics-related  strengths  and  weaknesses). 

2.  Discussion  of  the  organization's  strengths  and  weaknesses  based  on  each  of  the 
six  measurement  themes  identified  in  Appendix  A. 

b.  Discuss  recommendations  for  improvement  These  recommendations  will  be  based  on 

the  results  of  the  evaluation  and  may  include  one  or  more  of  several  elements,  such  as: 

1 .  A  recommended  set  of  high  payback  activities  that  the  organization  could  use 
to  implement  metrics  capability  improvements. 

2.  Recommendations  to  implement  a  metrics  improvement  program  that  would  be 
tailored  to  meet  the  specific  organization's  goals  based  on  follow-up  consulting 
and  plan  preparation.  These  recommendations  would  include  a  brief 
description  of  the  areas  to  be  covered  in  the  metrics  improvement  program  to 
help  open  communication  with  the  organization. 

3.  Recommendations  to  implement  other  management  and/or  engineering 
improvement  activities  that  would  be  tailored  to  meet  the  specific  organization's 
objective  based  on  follow-up  consulting  and  plan  preparation.  These 
recommendations  would  include  a  brief  description  of  the  areas  to  be  covered  in 
the  program  to  help  open  communication  with  the  organization. 

Figure  C-1  is  the  annotated  outline  for  the  software  metrics  capability  evaluation  report 
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1.  INTRODUCTION 

1.1  Identification 

Use  the  following  sentence  to  identify  the  evaluation  report:  "This  report  provides  the  results  of  a  software 
metrics  capability  evaluation  given  on  (review  dates,  in  mm/dd/yy  format)  for,"  then  provide  the  organization’s 
name,  office  symbol,  location,  and  address.  In  addition,  provide  the  approximate  size  of  the  organization 
appraised,  the  names  and  office  symbols  for  any  branches  or  sections  that  were  represented  from  within  a  larger 
organization,  the  basic  "type"  of  organization  (i.e.,  acquisition,  software  development,  software  maintenance),  and 
the  number  of  individuals  interviewed. 

1.2  Introduction  to  the  Document 

Identify  the  document’s  organization  and  provide  a  summary  of  the  information  contained  in  each  major  section. 

2.  APPRAISAL  RESULTS 

2.1  General  Results 

Give  the  metrics  capability  level  for  the  organization,  and  provide  backup  for  that  result. 

2JJ  General  Metrics  Strengths 

Provide  a  listing  of  general  areas  within  the  six  metrics  themes  represented  in  the  evaluation  where  the 
organization  showed  strengths,  e.g.,  establishment  and  general  use  of  a  metrics  database  or  general  examples  of 
management  decision  making  based  on  metrics  results. 

2. 7.2  General  Metrics  Weaknesses 

Provide  a  listing  of  general  areas  within  the  six  measurement  themes  represented  in  the  evaluation  where  the 
organization  showed  weaknesses,  e.g.,  no  metrics  database  or  identification  of  metrics  from  the  Air  Force  metrics 
mandate  that  are  not  being  collected  or  used. 

2.2  Specific  Areas  for  Improvement 

2.2. 7  Level  2  Areas  for  Improvement 

2,2J,X  Theme  X  Areas  for  Improvement 

For  each  of  the  six  measurement  themes,  provide  a  description  of  the  weakness(es)  for  that  theme.  Include  tlie 
following  topics  in  that  description: _ 


Figure  C-1.  Software  Metrics  Capability  Evaluation  Results  and  Recommendations  Report: 

Annotated  Outline 
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a.  Weakness(es) 

b.  Discussion 

c.  Reconunended  action 
2.2.2  Level  3  Areas  for  Improvement 
2.2.2.x  Theme  X  Areas  for  Improvement 

For  each  of  the  six  measurement  themes,  provide  a  description  of  the  weakness(es)  for  that  theme.  Include  the 
following  topics  in  that  description: 

a.  Weakness(es) 

b.  Discussion 

c.  Recommended  action 
3.  RECOMMENDATIONS 

Provide  any  general  recommendations  that  resulted  from  analyzing  the  appraisal  results,  e.g.,  need  to  determine 
general  management  approach  and  commitment  to  change  before  charting  a  detailed  metrics  improvement  plan, 
etc. 

Give  the  background  and  rationale  for  the  recommendations,  and  provide  a  set  of  positive  steps  the  organization 
could  take  to  improve  their  metrics  capabilities.  This  section  should  be  used  as  a  place  to  recommend  (or  propose) 
possible  first  steps  that  the  metrics  customer  and  the  STSC  could  explore  to  determine  whether  an  ongoing 
relationship  would  be  mutually  beneficial.  (In  the  case  of  metrics  capability  Level  1  organizations,  examples  are: 
to  undertake  a  study  of  the  organization's  culture  to  determine  the  easy  and  high  payback  activities  that  would 
give  the  organization  some  positive  results  for  minimal  effort,  to  work  with  the  organization’s  management  to 
determine  their  commitment  to  change,  etc.  Other  recommendations  could  include  working  with  the  STSC  or 
another  support  organization  to  develop  a  project  plan.) 

APPENDICES 

Appendix  A  contains  the  Measurement  Theme  and  Relationships  Table  (Table  A4  herein).  Also,  if  necessary, 
starting  with  Appendix  B,  provide  background  information  (e.g.,  the  customer  profile,  etc.)  that  would  be  difficult 
to  incorporate  in  the  main  body  of  the  report  or  that  would  interfere  with  the  readability  and  understandability  of 
the  evaluation  results.  _ _ _ _ 


Figure  C-1.  Software  Metrics  Capability  Evaluation  Results  and  Recommendations  Report: 

Annotated  Outline  (Continued) 
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APPENDIX  D.  ORGANIZATION  INFORMATION  FORM 

It  has  been  found  that  the  organization’s  culture  often  is  extremely  important  in  determining  how  best  to 
work  for  any  type  of  software  process  improvement,  including  establishing  a  working  metrics  program. 
This  appendix  has  been  developed  to  elicit  cultural  information  about  the  metrics  customer  that  will  help 
STSC  develop  the  project  plan  and  work  with  the  customer  for  their  metrics  capability  improvement. 

Credibility: 

1.  How  would  you  characterize  the  organization’s  customer  satisfaction? 

□  Excellent  □  Good  □  Fair  □  Poor 

Please  explain:  _ _ _ 

2.  How  would  you  characterize  the  organization’s  ability  to  meet  schedule  commitments? 

□  Excellent  □  Good  □  Fair  □  Poor 

Please  explain:  _ _ _ 

3.  How  would  you  characterize  the  organization’s  ability  to  meet  budget  commitments? 

□  Excellent  □  Good  □  Fair  □  Poor 

Please  explain:  _ _ 

4.  How  would  you  characterize  the  organization’s  product  quality? 

□  Excellent  □  Good  □  Fair  □  Poor 

Please  explain:  _ _ _ 

5.  How  would  you  characterize  the  organization’s  staff  productivity? 

□  Excellent  □  Good  □  Fair  □  Poor 

Please  explain:  _ _ 

6.  How  would  you  characterize  the  organization’s  staff  morale/Job  satisfaction? 

□  Excellent  □  Good  □  Fair  □  Poor 

Please  explain:  _ _ 

7.  How  frequently  do  the  development  projects  have  to  deal  with  changes  in  customer 
requirements? 

□  Weekly  or  Daily  □  Monthly  □  Less  Often  □  Rarely  if  Ever 

Please  explain:  _ 
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Motivation: 


1.  To  what  extent  are  there  tangible  incentives  or  rewards  for  successful  metrics  use? 

□  Substantial  □  Moderate  □  Some  □  Little  if  any  □  Don’t  know 

Please  explain:  _ _ _ 

2.  To  what  extent  do  technical  staff  members  feel  that  metrics  get  in  the  way  of  their  real 
work? 

□  Substantial  □  Moderate  □  Some  □  Little  if  any  □  Don’t  know 

Please  explain: _ _ 

3.  To  what  extent  have  managers  demonstrated  their  support  for  rather  than  compliance  to 
organizational  initiatives  or  programs? 

□  Substantial  □  Moderate  □  Some  □  Little  if  any  □  Don’t  know 

Please  explain:  _ _ _ 

4.  To  what  extent  do  personnel  feel  genuinely  involved  in  decision  making? 

□  Substantial  □  Moderate  □  Some  □  Little  if  any  □  Don’t  know 

Please  explain:  _ _ _ 

5.  What  does  management  expect  from  implementing  metrics? 

Please  explain:  _ _ 
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Culture/Change  History 

1.  To  what  extent  has  the  organization  used  task  forces,  committees,  and  special  teams  to 
implement  projects? 

□  Substantial  □  Moderate  □  Some  □  Little  if  any  □  Don’t  know 

Please  explain:  _ _ _ 

2.  To  what  extent  does  “mrf  guarding”  inhibit  the  operation  of  the  organization? 

□  Substantial  □  Moderate  □  Some  □  Little  if  any  □  Don’t  know 

Please  explain:  _ _ _ 

3.  To  what  extent  has  the  organization  been  effective  in  implementing  organization 
initiatives  (or  improvement  programs)? 

□  Substantial  □  Moderate  □  Some  □  Little  if  any  □  Don’t  know 

Please  explain:  _ _ _ 

4.  To  what  extent  has  previous  experience  led  to  much  discouragement  or  cynicism  about 
metrics? 

□  Substantial  □  Moderate  □  Some  □  Little  if  any  □  Don’t  know 

Please  explain:  _ _ _ _ 

5.  To  what  extent  are  lines  of  authority  and  responsibility  clearly  defined? 

□  Substantial  □  Moderate  □  Some  □  Little  if  any  □  Don’t  know 

Please  explain: _ _ _ 
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Organization  Stability 

1 .  To  what  extent  has  there  been  turnover  in  key  senior  management? 

□  Substantial  □  Moderate  □  Some  □  Litdeifany  □  Don’t  know 

Please  explain: _ _ _ 

2.  To  what  extent  has  there  been  a  major  reorganization(s)  or  staff  down-sizing? 

□  Substantial  □  Moderate  □  Some  □  Litdeifany  □  Don’t  know 

Please  explain: _ _ _ 

3.  To  what  extent  has  there  been  growth  in  staff  size? 

□  Substantial  □  Moderate  □  Some  □  Litdeifany  □  Don’t  know 

Please  explain: _ _ _ 

4.  How  much  turnover  has  there  been  among  middle  management? 

□  Substantial  □  Moderate  □  Some  □  Litdeifany  □  Don’t  know 

Please  explain: _ _ _ 

5.  How  much  turnover  has  there  been  among  the  technical  staff? 

□  Substantial  □  Moderate  □  Some  □  Litdeifany  □  Don’t  know 


Please  explain:  _ _ _ 

Organizational  Buv-In 

1.  To  what  extent  are  organizational  goals  clearly  stated  and  well  understood? 

□  Substantial  □  Moderate  □  Some  □  Litdeifany  □  Don’t  know 

Please  explain:  _ 

2.  What  level  of  management  participated  in  the  goal  setting? 

□  Senior  □  Middle  □  First  Line  Mgt  □  Don’t  Know 

Please  explain:  _ _ _ 
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3.  What  is  the  level  of  buy-in  to  the  goals  within  the  organization? 

□  Senior  Mgt  □  Middle  Mgt  □  First  Line  Mgt  □  Individual  □  Don’t  know 

Contributor 

Please  explain: _ _ _ 

4.  To  what  extent  does  management  understand  the  issues  faced  by  the  practitioners? 

□  Substantial  □  Moderate  □  Some  □  Litdeifany  □  Don’t  know 

Please  explain: _ _ _ 

5.  To  what  extent  have  metrics  been  used  for  improving  processes? 

□  Substantial  □  Moderate  □  Some  □  Litdeifany  □  Don’t  know 

Please  explain: _ _ _ 

6.  To  what  extent  has  there  been  involvement  of  the  technical  staff  in  metrics? 

□  Substantial  □  Moderate  □  Some  □  Litdeifany  □  Don’t  know 

Please  explain:  _ _ 

8.  To  what  extent  do  individuals  whose  work  is  being  measured  understand  how  the 
metrics  are/will  be  used  in  the  management  process? 

□  Substandal  □  Moderate  □  Some  □  Litdeifany  □  Don’t  know 

Please  explain:  _ 
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Measurement  Knowledge/Skills 

1.  How  widespread  is  metrics  knowledge/training? 

□  Substantial  □  Moderate  □  Some  □  Little  if  any  □  Don’t  know 


Please  explain:  _ 

2.  What  type  of  metrics  training  have  members  of  the  organization  participated  in? 


□  Statistical  □  Data  Analysis  □  Metrics  □  Basics 

Process  Application 

Control 


□  Don’t 
know 


Other: 
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